this is very much a work in progress rough draft, but if anyone has suggestions for tips/tricks i should cover please feel free to make a suggestion @fritzo @eb8680 @fehiepsi @ordabayevy

Looks great! Here are some comments:

This is great!

Some comments on some existing sections:

• Introduction: add a plug and link for the forum e.g. "If you’re having trouble finding or understanding anything here, please don’t hesitate to ask a question on our forum!"
• Learning rates: add a sentence explicitly justifying/recommending Adam ("try optim.Adam by default because ..."), point to schedulers
• Autoguides/custom guides: add a sentence at the end of one of these sections mentioning easyguide with a link to the easyguide tutorial

Some suggestions for new sections:

• as @fritzo mentioned, enabling validation (pyro.enable_validation(True) and *ELBO(strict_enumeration_warning=True)) to help catch silent shape/support errors or track down nans - this is covered somewhat in the tensor shape tutorial but probably worth teasing here and telling people to go read the tensor shape and enumeration tutorials
• basic tools for reducing variance: use more particles via num_particles=..., increase batch sizes, use gradient clipping or learning rate schedulers/decay, use enumeration or TraceMeanField_ELBO if applicable, try annealing the prior (with link to DMM example), etc.
• perhaps out of scope for this tutorial, but you could suggest a high-level pattern for code organization e.g. making model and guide separate PyroModules and combining them in a top-level nn.Module with methods for training and prediction

It looks great!

Some practical suggestions:

1. Checkpointing. This is mentioned in https://pyro.ai/examples/dmm.html for nn.Module. The snippet of code on how to do it using ParamStore:

def save_checkpoint(path_param, path_optim):
    # save only if there are no NaN or Inf values
    for k, v in pyro.get_param_store().items():
        if torch.isnan(v).any() or torch.isinf(v).any():
            raise ValueError(f"Detected NaN or Inf values in {k}")

    # save parameters and optimizer state
    pyro.get_param_store().save(path_param)
    self.optim.save(path_optim)

def load_checkpoint(path_param, path_optim):
    pyro.clear_param_store()
    pyro.get_param_store().load(path_param, map_location=self.device)
    self.optim.load(path_optim)

2. Brief guide on how to implement new distributions. This might deserve its own tutorial on how to implement & test new distributions. But maybe at least mention that Pyro distributions need to subclass TorchDistribution or TransformedDistribution and point to TorchDistribution which has a small section on how to implement Pyro compatible distributions.

Brief guide on how to implement new distributions

I'm vaguely planning to created a dedicated tutorial on implementing custom distributions. Before that, I'd like to create more automated testing tools along the lines of goftests #2658.

@fritzo @eb8680 @ordabayevy thanks for your feedback! i've tried to incorporate what seemed in scope (basically things related to optimization).

this is now ready for review. please check links and let me know if you can think of additional content that might be worth adding that's in scope

Final comments:

• 3. Add a comment on gamma like

  gamm = 0.1  # final learning rate will be 0.1 * initial learning rate

• 4. Add a line comment on bad_guide() like

  pyro.sample("x", dist.Normal(loc, 1.0))  # Normal may sample x < 0

• 8. Mention that "To use vectorized_particles=True you may need to ensure your model and guide support batching. See the tensor shapes tutorial for best practices."

thanks @fritzo addressed your comments. this should be good to merge unless @eb8680 has additional feedback